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This summary of the 1973-74 Hichigan 
Cost-Effectiveness Study describes the results of an exploratory 
inquiry into compensatory education reading programs in forty-eight 
Michigan schools. The study is said to have as its purpose the 
development and implementation of evaluation techniques to determine 
what educational practices bring about changes in students* behavior 
and what costs are associated with these. On site visits to each of 
the scljools provided measures of program cl^aracteristics, including 
cost data. Analysis of the data included a comparison of program 
characteristics with 1972-73 and 1973-74 student reading achievement. 
"Controllable" program characteristics, said* to describe the 
difference between effective and non-effective compensatory reading 
programs, are indicated. These are a set of controllable factors 
-^ela:-ted to student reading achievement arid pertaining to school 
administrators and teachers, and those d,enoting that, when other 
things were equal and up to a point, more dollars per student means 
more achievement per student. Continuation of the program for 1974-75 
is suggested with the following foci: identification of new variables 
that relate to achievement.; extension of relationships between cost 
and achievement; and, investigation of -fehe direction of the 
relationship between achievement and the various identified 
variables. (Author/AM) 
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GENERAL INFORMATION SHEET 



Enclosed is a copy of the report sent to the State Board of Education 
entitled, "Executive Summary of the 1973-74 Michigan Cost„Effectiveness Study". 
As you will note, the Summary is dated February 25th, having been prepared by 
the Division Research, Evaluation, and Assessment, Michigan Department of 
Education several months prior to that date. Hence, this Summary reflects 
a condensation of the September 30th Report submitted by Education TURNKEY 
Systems to the MDE, and does not include much analysis and re-analysis con- ^ 
ducted -by both the MDE and' Education TURNKEY Systems after December 1974. 
Since the discussion of the report on March 5th included the release of the 
results of subsequent analysis and re-analysis ,_a ^ress release highlighting 
some of the major subsequent findings is also included. 

For additional information, it is recommended that interested parties 
contact either Dr. Michael Hunter, Division Research Evaluation & Assessment, 
Michigan Department of Education, Lansing, Michigan (517) 3-'3-1830; or Mr. 
Charles Blaschke, Education TURNKEY Systems, fnc, 1660 L Street, N. W. , 
Washington, D. C, 20036, (202) 293-5950. 



1 

i 



Enclosure 



US DEf>ARTMENTOf^ HEALTH, 
EDUCATIONS WCLPARE 
NATIONAL INSTITUTE OP 
EDUCATION 

rM«S OOCv\f<ENT MAS eCCN PEPPO 

O'jcco exactly as RECCiveo r^ROM 

The person op OROANfZATiO^^OffiOiN 
AtfNO »T POfNTSOf V»eW0P OP'NiOfJs 
STATED DO NOT NECESSAPav PEPPt 
SENrO^^*C*AL MAl*ONAt iN$T»TyTE qp 
EO'KATTON POSiTiON 0« POLffY 



ERIC 



2> 



FOR DISCUSSlOrONLY 



Statement Byi 
HJCHIGAN DEPARTMENT OF EDUCATIOM 
EDUCATION TURNKEY SYSTEMS, INC. 

Regarding: ^ 
HI CHI GAM DEPARTMENT OF EDUCATION 
• RELEASE OF RESULTS OF 
.C0HPENSAT(3RY EOUCAT-IOH COST-EFFECTIVENESS STUDY 

(5 H!VRCH, 1975)- 
DURING BOARD MEETING 

Too often, education research and evaluation efforts "recommend 

further" study end analysis to continue searching for elusive variables which 

might explain differences In student outcotncs. In this case, the opposite- Is 

true. Th© results of this exploratory study must be extended for cross-. 

validation purposes because of the large number of statistically significant 

variables which have been Identified as characterlstlc-of successful comp-ed 

reading programs, and the serious challenges to assertions that schools do 

not affect student achievement. Yot, with guarded caution and optimism, we 

feel strongly that the evidence presented In our September 30th report and 

the March 5th sunmary report to the Michigan State Board of EducatlonJ 

• refutes the thesis that schools don't make a difference 

In Student performance, while Identifying specific variables 

(and their respective costs) which do appear to make a 

difference. 

"1 establishes a new methodology for Identifying the Impact ^ 

of schools and resources on students performance. 
, corroborates the observation of experienced administrators 
and teaching staff that classroom management and decision- 
making variables at the building level are critical to 
.. program success. ' ^ 



, Indicates that there are variable* which make a difference ' 
'v . and are controllable by local district policy makers. 

The inajor limitation to the general Izeablllty of the study's 
findings fc6 « national audience could be the unique leadership and priorities 
of the Michigan Department of Education and the Ingenuity and perseverance 
of Michigan comp-ed district staff at the local level. The policy Implications 
of this study are significant Indedd, at local, state, and federal levels; 
yet as noted below, they are tentative, pending further, Indepth analysis 
of the~data to confirm the findings, a rather awesome responsibility In light 
of the. study's implications. 

First, while not Identifying specific causal relationships, the study 
does Indicate that some schools (programs) In Michigan do make a difference 
and the characteristics of these schools are In many Instances very significant. 

, Second, the factors which appear to discriminate. If not account for, 

the difference between effective and non-effective comp-ed programs, are for 
the most part "controllable" by local district staff, and are usually those 

at the building level: . 

, The classroom monitoring role of the principal end how he 
allocates his time and delegates decision-making over certain 
factors to the classroom teachers. 
♦ The role of teachers, the degree of decision-making delegated 
to them, and the amount of time the teachers allocate to- 
Instruction management activities Including diagnosis and 
■ prcsc/lptlon, and the development of performance objectives 
for Individual students. 



. The nature and extent of coordination among building staff, 
vspecUUy between tho cotnp-ed teacher and the regular 

t ' 

teachers. 

, Tlje amount of time allocated by the comp-ed director and 
teachers to planning and pre-scrvlce training. 

While further analysis will undoubtefily provide more Insights, district 
staff now have some useful Information on Implicit actions/Initiatives which 
can be used to Improve their programs. 

Third, the Implications for .funding allocation to comp-ed programs are 
difficult to assess without further analysis. It Is clear that a large amount 
of resources (e.g., nearly $670/pupll In comp-cd reading In aTV study programs) 
are allocated to comp-ed reading programs, perhaprr^latlvely more than In other 
states, though comparable data are not truly available since other states have 
not used the cost methodology used here. Moreover", successful programs allocate 
«orc than do not-so-5uc*<^essful ones (I.e., $7'»2 In high vs $587 In low per pupil). 
And the amount of resources allocated to comp-ed reading are highly correlated 
to student reading achievement, by itself explaining over 30S; of the observed 
variance In this achievement. And finally, other things being equal and up 
to a po.lnt, more resources allocated per s.tudent to reading means more 

achievement per student. --^ 

While on the surface. It appears that Title I programs allocate more 
resources than Chapter 3 (i.e.. the state-funded $r2.5 million student performance 
pact program) some of the differences could bo related to tho nature of 
-guidelines, Us history, or the different types of Incentives available to tho 
districts* 



In terms of the composition of cocnp-ed resource utilization, one 
specific Implication arises: the factors or variables which characterize the . 
differences cost few additional marginal dollars; rather they reflect different 
allocation of existing sta'f time of building personnel* 

A$ future analyses and^ro-analyses of the existing data are conducted 
over the next few nwnths and. cross-vaWdatlons using 197^-75 data occur, policy 
Implications will be addressed with greater specificity. However, this study 
Involving districts conducted by the Michigan Department of Education and 
Education Turnkey Systems, Inc. corroborates the recent findings of other , 
recent studies (Federal Reserve Board, Philadelphia, 1975 and State of New York, 
Governor's Office, .137^), thus providing some direction, If not supportive data, 
for education policy makers at federal, state, and local levels* ' 
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TO: 



MICHIGAN DEPARTMENT OF EDUCATION 
MEMORANDUM 

Members of State Board of Education DATE:- February 25, 1975 



FROM: John W. Porjier 

SUBJECT: Information on Compensatory' Education 
' Cost-Effectiveness Study 

, On November 6, 1973 the State 3oard of Education approved a contract 
between the Department and Educat ion. Turnkey Systems, Inc. for the purpose 
of conductinq the 1973-?^ Michigan Cost-Effectiveness Study. The purpose 
of the study was to continue development and begin implementation of cost- 
effectiveness procedures developed by Department staff. 

The study focused on compensatory education reading programs. Porty; 
* eight schools were selected. Measures of program characteristics, including 

cost data, were obtained during on-site visits to each of the schools. The 
various analyses included th^ comparison of program characteristics to both 
1972-73 and 1973-7^ student reading achievement. o 

O Without identifying specific causal relationships, the study indicates 
. that some program characteristics are systematically related to student 
reading achievement. 'These characteristics which appear to describe the 
differences between effective and noneffective compensatory reading programs 
are,' for the most part, "control 1 abl e" -by local district and often school 
building staff. While further analysis will provide more insights, this 
study provides district staff some useful information on implicit actions/ 
• initia-tjves whi.ch might be used to improve reading programs. 

The implications for funding allocation, to compensatory reading pro- 
' V grams are dlfflcult-to assess without further analysis. Large .amounts of 
resources are allocated to compensatory reading programs in Michigan. It 
was found that successful programs spent more money than not-so-successful 
programs. However, other factors also influence program effect i veness . 

Briefly, the study has identified: 

1. A set of controllable factors that are related to student reading 
achievement. These factors pertain to both school administrators 
and teachers. 

2. That other things being equal and up to a point, more dollars per 
student means more achievement per student. x 

It Is recommended that the State Board of Education receive report on 
the 197W^ Michigan Cos-t-Ef feet i veness Study: Executive Summary . 

ERIC . 
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MICHIGAN COST-EFFECTIVENESS 
STUDY: AN EXECUTIVE. SUMMARY- 



December, 1974 



Michigan Department of Education 

Research, Evaluation, anci 
Assessment Services 
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INTRODUCTION 



The purpose of this summary is to provide a description of 
the 1973-74 Michigan Cost-Effectiveness Study and its findings. The 
study was an effort to develop and implement evaluation techniques 
which can determine what educational practiced bring about changes 
In stujient behavior and what costs are associated with those practices 
'^TUfie 1973-74 study was restricted to compensatory education reading 
programs. 

The term program effectiveness, as used in this study, includes 
a consideration of both program success and activities associated with 
the^progr'am. Program success is attained when the objectives of a 
program are attained, i.e., an increase in student reading achieve- 
ment. Program effectiveness required further" investigation of the 
question, "Why was the program successful?'* To be termed effective, 
the activities of the program must contribute to the success of the 
program so that there is a strong indication that the activities 
brought about the achievement of the objectives. * ^ 

/ 

Successful programs must be examined to determine their effec- 
tiveness. However, a problem arises when only successful /programs 
are examined. For example, if af group of successful individualized 
instruction programs showed that all successful programs collect 
student attendance data, it might be reasonable to assume that 
collection of student attendance information was one reason for the 
success of the programs. However, an examination of unsuccessful 
individualized instruction programs would most likely show that 
they too collect student attendance data. It would be erroneous 
to attribute program success to the collection of attendance data. 
Thus, it is necessary to examine both successful and unsuccessful 
individualized instruction programs in order to determine what ^ 
activities are present in successful programs, but not in unsuc- 
cessful programs. \ ^ 

SITE SELECTION 

As a first step toward identifying effective educational 
practices, thirty-three unusually successful and thirty-three 
unusually unsuccessful compensatory education delivery systems were 
randomly selected. These delivery systems were first identified 
as programs at the school district level and then as programs at 
schools within the school district. 

Through written corxespondence and telephone contact, screening 
criteria were applied to the sixty-six final program sites. These 
were: 
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!• The compensatory education project (Title I or Chapter 3) 
was in existence by the fall of 1972. 



The project had the saiue key project person (e.g., reading 
coordinator) in 1973-74 as in 1972-73; or the same key 
person provided the same services to the project as was 
provided the previous year, even though this person might 
hold a different title or be. in a different "location. 

3^^ ^The school building had the same principal in 1973-74 as 
in^i972-73. 

4. Teacher and student turnover in the building was less 
than 40%. ^ 

5. There were at least five students per participating grade 
level. . . 



6. The program materials used>were essentially the same in eacf 
of the two years. 



Forty-eight final sites ,\ consisting of twenty-five high achiev- 
ing schools and twenty-three jLow achieving schools were identified. 
Table 1 presents. a breakdown 



Df the final data sites. 

TABLE 1 

FINAL SITE ^ELECTION FOR 
* MICHIGAN COST-EFFECTIVENESS STUDY 



a 


TITLE I 


CHAPTER 3 


TOTAL 


High Achieving Districts 


18 


7 


25 


liijw Achieving Districts 


17 


6 


.23 


' TOTAL 


^ ^5 


13 


48 



Achievement data from the 1973-74 test* results for the Title I 
and Chapter 3 Compensatory Education Programs in tHese schools were 
collected and compared to the results from. 1972-73. The month/ 
month gain scores were computed for the schools visited during the 
1973-74 school year. Table 2 illustrates'>;the degree of shifting 
from the 1972-73 ranking within- a group. Plor instance, the highest' 
ranking Title I site for 1973-74 was previously ranked fifteenth 
for the 1972-73 school year. However, a high degree of stability 
was exhibited in so far as a site retaining its original classi- 
fication of being a high or low achieving district. Only six 
sites actually changed from one group to another. 



TABLlS 2 



1973-74 READING RESULTS FOR TITLE I AND CHAPTER 3 
SITES BY HIGH AND LOW GROUPS 





HIGH 




LOW 




I 

Rank 


Position 


Month /Month 


Rank Positioti 


Month/Month 


1972-73 


1973-74 


Gain 


1972-73 1973-74 


Gain 



Title I 



1 


2 


2.38 


19 


30 


.88 


z 


7 


1.88 


20 


27 


.97 




14.5 


1.34 


21 - 


21. 


1.11 


.A 


ft 


1.85 


22 


18 


1.31 


5 


13" 


i.41 


23 ■ 


24.5 


1.04 


6 


4 


?.02 


.24 


32 


.83 


7 


11 


1\.50 


, 25 


31 


.86 


8 


14.5 


li34 ■ 


26 


33 


-.76 


9 


12 ■ 


l.\43 


27 


26 V 


•1.01 


10 


'6 


1.94 


28 


2=3 


1.08 


11 


17 


1.33 


^ 29 


22 


1.10 


12 


19 . 


1.22 


30 


^8 


.96 


13 


5 


1.95 


31 


.20 


1.13 


14 


24.5 


1.04 


32 


29 


.93 


15 


1 \ 


3V62 


33 


34 


.54 


16 


16 \ 


li33 " 


34 


:9 


1.66 


17 


10 


1.51 


35 




.50 


18 


3 


2.10 










\ 


Chapter 3 






1 


6 . 


1.23 


8 


13 


.79 


2 1 


3 


1.56 


9 


10 


.82 


3 


5 


1.40 


10 


7 


1.08 


A 


2 


1.66 


11 • 


8 


1.03 


5 


4 


1.50 


12 


9 


■ .86 


6 


11 


.81 


; 13 


12 


.80 


7 


1 


2.19 
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DATA COLLECTION 

\ 

The first step in the data collection process was to develop 
instruments. A preliminary data gathering instrument had been 
developed and field tested during May arfd June of the 1972-73 school 
year in fifteen projects in eight Michigan school districts. This 
instrument was reviewed by Education Turnkey Systems, Inc. (TURNKEY) 
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staff and consultants. Nine major groups of data items and four 
respondent levels were identif^Led. The'-data item groupings included: 

* • 

1. Staff Variables ' ■ 

2. Organization and Management o^ Overall Program 

3. Organization and Management of«£Lasaroo°i Reading Activities 
4» Method. of Instruction 

5. Staff Development" . ' 

6. Student Characteristics , ' . . 

7. School/District Characteristics * , . ' 

8. Utilization of Staff Time / 

9. Participants (Students, Staff, Parents, and Others) 

Specific items under each category were developed by TURNKEY 
staff tased on review of the literature ahd/or adapted from previ5)usly 
existing instruments. Appendix A provides a source reference for . 
the -i'tem pool used in the instrument development and revision process. 

\Five respondent levels we're identified. These were: 

1. The district director of compensatory education 

'2. The target school principal • . ' 

"21, ITie compensatory education reading teacher (s) 

4. The regular reading teacher(s) / 

5. Other staff. , . • 

. Both regular reading teachers and special compensatory education 
teachers responded to the same teacher questionnaire form. Reading 
specialists, reading coordinators, and paraprofessional reading axdes 
responded to the questionnaire form entitled "Other . This Other 
form was also designed to obtain information from additional staff 
that had direct involvement with the compensatory education reading- 
resource teacher paid by Title I or Chapter 3 funds. Items for the 
final draft of the instrument were included on the basis of the 
degree -to which they reflected factors. of cost and program para- 
meters^that can be controlled via policy decisions. 

The^ data collection team consisted of one data\manager and nine 
data colleptors. The persons on the data team had ha^ prior teach- 
ing experience or were graduate level studepts in education. 

In February 1974, prior to .the initiation of the data collection 
process, TURNKEY staff conducted a two-day ihLansive orientation and 
training session for the data collectors. In addition, KDE oxficials 
attended the session. • ^ 

Data collectors were provided only that informaticJn and train- 
ing which they would need to fulfill pheir specific responsibilities. 
For example, while the general design of the evaluation effort was 
described, certain factors which could have affected the nature of 
the data collection were carefully avoided in the training (e.g., 
that sites were selected on the basis of high or low achievement). 
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The typical data collector visited an average of seven sites 
with the typical site consisting of one director, one principal, 
one compensatory education teacher, three regular teachers, and 
two paraprofes^ionals. Individual interviews lasted approximately 
fifty-five mir\utes over an 8.5 hour -interviewing day. Data 
collectors; reported that respondents were "cordial, helpful, and 
cooperative . . .with every effort being made to provide the 
requested information". 

' data' ANALYSIS 
* * • 

The effectiveness analysis began with the nine major groups 
of data items which', it was hypothesized, would act together in their 
impact on reading program effectiveness. Initial phases of the 
effectiveness analysis dealt with identifying those groups of items 
which could discriminate betwjeen the high and low schools. Of the 
719 items (not including cost, items), approximately 43.5 items were 
included in the analysis,. The remaining items were excluded for 
various reasons such as incomplete data, all schools responding 
in a like manner, no s^chools, responding, etc. 

"Of the 9 groups of items, only the building principals . 
responses to the group concerning student,''^ parent and staff involve- 
ment in the project discriminated between high and low schools. 
However, 45 of the individual items discrijainated betv7een high 
and low schools. Since high and low schools were selected from 
the basis of 1972-73 reading achievement results, .the 45 variables 
were cross validated to the 1973-74 reading achievement results. 
The Table 3 presents results Qf this cross validation which showed 
that 17 of the 45 variables maintained their relationship to achieve- 
ment over two years. , ^ 

The type of relationships between variables in Table 3 and 
reading achievement is shown in the right column of the table. For 
example, those schools where the principal was satisfied with 
compensatory education curriculum decision methods had significantly 
higher reading achievement than did those schools where the principal 
was not satisfied with compensatory education curriculum decision 
methods. Similarly, where the fraction of materials selected by 
the compensatory education teacher was high, reading achievement was 
high and vice versa. 

The relationships presented in Table 3 represent the major 
product of the 1973-74 study.' Jhey ar;e the current best indicators 
of what is related to student reading achievement. For example. 
Table 3 shows that when a compensatory education director spent 
more than an average amount of time planning compensatory reading 
programs, the student reading achievement , was greater than when 
a i^irector spent less than the average amount of time planning • 
In a strict sense,, it is not known, howeviir, whether more pla^nning 
brought about higher achievement or higher achievement brought 
about more planning. Common sense suggests the former. 
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TABLE 3 



VARIABLES -RELATED TO READING ACHIEVEMENT 



Variable 



Type of Relationship 



District Coordinator (respondent) 

Percent time planning comp. ed» reading 



Direct, positive 



Principal (respondent) 

Is principal satisfied with comp* ed. curri- Yes: 

culim decision methods? Direct, positive 



Number of teacher working hours at school 
daily. 



Direct, positive 



Comp. Ed* Teacher (respondent) 

Fraction of materials selected by teacher. 

Were periodicals basic reading materials? 

Days of training provided teachers at r 
onset of projects. 

Degree to whic}i comp. ed. students liked 
school in 1972-73. 

Does teacher know percent of comp. ed. 
students absent from classroom on 
any given day? 

Did paraprofessionals help teacher? 



Teacher morale 



D-^rect, positive 
Yes: 

Direct, positive 
Direct, positive 



Direct, positive 

Yes: 
Direct, postive 



No: 

Direct, negative 
Direct, positive 



Regular Classroom Teacher (respondent) 

Number of classroom observations by reading 
specialist over last 12 months 

Was non-professional tutorial part of the 
1972-73 combination project (only teachers 
from combination type projects responded)? 

Percent of time professional tutorial was 
part of combination projects in 1972-73. 



Direct, positive 



No: 

Direct, negative 



Indirect J negative 
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TABLE 3 (con't.) 



VARIABLES RELATED TO READING ACHIEVEMENT 



Variable 



Type of Relationship 



Regular Classroom Teacher (respondent) (con'tO 

Percent professional tutorial was 

part of combination' projects in 1973-74 



Indirect , negative 



Were commercial reading texts supplementary? 



Yes: 

Direct, positive 



Difficulty of reading materials 



Indirect, negative 



Percent of time spent (by teacher) on mis- 
cellaneous (other than instruction, reading, 
or administrative activities. 



Indirect, negative 



/ 



Likewise, it is not known whether higher teacher morale results 
in higher student reading achievement or higher student reading 
achievement results in high teacher morale. What is currently known 
is that the morale of compensatory .education reading teachers is 
directly related to student reading achievement. The question of 
which of the variables in Table 3, and other variables not examined 
during 1973-74, 'effect student reading achievement is being further 
addressed by the 1974-75 phase of the Cost-Effectiveness Study. 



The COST-ED Model, applied by TORNKEY in the analysis of educa- 
tion programs throughout the United States was adapted for use in 
the analyses of the costs of the forty-eight compensatory educa- 
tion, projects included. in this study. Each program was modeled as 
being made up of one activity in which the student was involved 
(classroom reading activities) plus four supportive activities not 
involving the students* time directly (planning, training, decision 
making, and administration). 

The average per pupil cost of the project for each of ^orty-eight 
sites was determined. Only costs attributed to compensatory reading 
instruction were included, e.g., the portion of the district director's, 
principal's, compensatory education teacher's, regular classroom 
teachers'., and other staff's time for this instruction was calculated 
and applied to the total cost for compensatory reading instruction. 
These data are shown in Table 4 and 5. The figures in Table 4 and 5 
represent the average, cost of reading instruction, by various categories 
for compensatory education students. The costs are comprised of 
the sum of the cost borne by the compensatory education program, the 



COST ANALYSIS 
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cost borne by other fund sources, and the estimated dollar value 
of contributed resources. An example of the latter would be the 
estimated dollar value of unremunerated services provided by teachers, 

The Tables show that the high achieving sites averaged $742.00 
per student while the low achieving sites averaged $587-19 per 
student* Only in the area of administrative functions did the low 
achieving sites spend more money than the high achieving sites. For 
both high and low achieving sites, more money was spent for reading 
instruction than for any^ other function. 
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CONCLUSIONS 



Without identifying specific causal relationships, the study 
Indicates that there are certain program characteristics which are 

^stematically related to student reading achievement • These charac- 
t^istics which appear to describe the differences bctvzecn effective 
an\ non-effective compensatory reading programs are for the most part 
"controllable" b^y local districts and, often, school building staff. 
While further analysis will provide more insights, this stjudy pro- 
vides district staff some useful information on implicit a-ctions/ 
initiatives which! might be used to improve reading programs. 

1 * 
The implications for funding allocation to compensatory reading 

programs are difficult to assess wijthout further, analysi'^ • Yarge 

amounts of resources are allocated to cpmpensatoiry r.e^</ing programs 

In Michigan. It was .found that successful programs 'spend more >. ^ 

money than not-so-suc^ssful programs. However, other factors ajtso 

influence' program effectiveness* - \ 

Briefly, the study has identified: i*. " ^ 

1. A set of, controllable factors that ai^e related to student 
reading achievement. These factors pertain to both school 
administrators and teachers, and 

2. That other things being equal and up to a point, more 
dollars pier student means more achievement per student. 

■ v' " • 

To achieve full benefit of the 1973-74 study, It should be 
continued in 197A-75. The continuation should: 

\ 

1. Identify new variables which relate to achiev^ement , 

2; Extend relationships between cost and achievement, .and 

3?^ Investigate the direction of relationship between achieve- 
ment and the various identified variables. 



\ 
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